Discrimination Analysis of Lip Motion Features for Multimodal Speaker Identification and Speech-reading

In this thesis a new multimodal speaker/speech recognition system that integrates audio, lip texture, lip geometry, and lip motion modalities is presented. There have been several studies that jointly use audio, lip intensity and/or lip geometry information for speaker identification and speech-reading applications. This work proposes using explicit lip motion information, instead of or in addition to audio, lip intensity and/or geometry information, for speaker identification and speech-reading within a unified feature selection and discrimination analysis framework, and addresses two important issues: i) Is using explicit lip motion information useful? and ii) if so, what are the best lip motion features for these two applications? The best lip motion features for speaker identification are considered to be those that result in the highest discrimination of individual speakers in a population, whereas for speech-reading, the best features are those providing the highest phoneme/word/phrase recognition rate. The audio modality is represented by the well-known mel-frequency cepstral coefficients (MFCC) along with the first and second derivatives, whereas lip texture modality is represented by the 2D-DCT coefficients of the luminance component within a bounding box about the lip region. Several lip motion feature candidates are considered including dense motion features within a bounding box around the lip, lip contour motion features, lip shape features, and combinations of them. Furthermore, a novel two-stage discriminant analysis is introduced to select the best lip motion features for speaker identification and speech-reading applications. The fusion of audio, lip texture and lip motion modalities is performed by the so-called Reliability Weighted Summation (RWS) decision rule. Experimental results show that the proposed discriminative analysis significantly improves the unimodal performance of the lip motion modality. Moreover, using explicit lip motion information in addition to audio and lip texture yields further performance gains in bimodal speaker/speech recognition systems.